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METHOD FOR EVALUATION AND STABIL 

CLASSIFICATION RESULTS 



iTZATION OVER TIME OF 



BACKGROUND INFORMATION 

The present invention relates to a method and system for 
evaluating classification results from classification methods 
which proceed in computer-assisted fashion and in which 
5 objects to be classified are sensed using sensors over a 
period of time, and are repeatedly classified with the 
inclusion of specific quality parameters for an object class. 
A quality parameter of this kind is a parameter number or 
another qualifying indication within a preselected range, and 
10 is expressive of the certainty that the object belongs to the 
particular object class and not to a different object class. 

Observations of objects are evaluated today in many 
application sectors, e.g., in machine vision in industrial 

15 production processes, in safety-related applications, for 

sensing the surroundings of vehicles, etc. These observations 
are furnished by measurement devices that can comprise, for 
example, one or more sensors that in some cases is different, 
e.g., radar, ultrasound, video in the visible and infrared 

2 0 region, lidar, laser, range imager. 

One goal when processing object data obtained in this fashion 
is classification. "Classification" is understood to be the 
allocation of objects to specific object classes. A class 

25 serves to describe multiple objects that are considered, on 
the basis of similar features, to belong together. 
Classification systems for the assignment of objects to object 
classes are used, for example, in industrial production. 
Sensing and object classification of the surroundings of a 

30 motor vehicle while driving also promises great benefits. 
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Classification according to conventional classification 
methods has the disadvantage that, especially in borderline 
cases with regard to allocation of the sensed object to a 
5 class, an elevated risk of misclassif ication exists. In 

borderline cases, for example, there is a high probability 
that in the context of individual classification results, the 
object will be attributed to the wrong object class. Outliers 
in the context of classification of an object, resulting, 
10 e.g., from measurement errors, also have a negative effect on 
the stability and robustness of conventional classification 
methods . 

The conventional methods furthermore offer only a yes/no 
15 classification of the sensed object. These results are 

insufficient for situations that require a more differentiated 
classification result, for example in the context of 
estimating accident hazards for a motor vehicle. One reason 
for this is that in the context of a classification repeated 
20 over time, an object is allocated first to one class and then, 
as the object is tracked, later on to a different class; this 
instability means that a clearly interpretable result is not 
supplied. Information as to how certain it is that the object 
might not also belong to one of the other classes would 
25 therefore be additionally important. 

SUMMARY 

In accordance with example embodiments of the present 
invention, a stabilization over time of the classification may 

30 result. According to the present invention, a confidence 

parameter is calculated from the quality parameters of the 
individual measurements, and is generally "integrated" over a 
certain number of time steps k and corresponding individual 
measurements. An abruptly large change in the confidence 

35 parameters for each object class with reference to successive 
time steps is thereby avoided, so that individual outliers 
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during classification can be filtered out. A strengthening or 
weakening of the classification result is also effected when a 
subsequent classification of the result confirms or does not 
confirm a previous one. "Confirmation" is understood here as 
5 at least a tendency toward allocation of the object to the 

same class. For example, a classification with reference to an 
object class as the third most suitable class among ten 
classes can also represent a confirmation for a previous 
classification of the object in that class. 

10 

Because, according to an example embodiment of the present 
invention, the method continuously repeats, the classification 
result in borderline cases is stabilized to the somewhat more 
probable object class, so that disruptive back-and- forth jumps 
15 in the classification result are suppressed and the method is 
robust in terms of such disruptions. 

The utilization of time-related information in the context of 
the present invention furthermore makes possible a more 

20 differentiated evaluation of the classification result. This 
kind of "final classification result, " based on time-related 
context information and as a function of confidence parameters 
adapted by way of many measurements, can yield a substantially 
better result than a classification based on individual 

25 results. 

One advantageous refinement of the example method according to 
the present invention is a dependence of the increase and/or 
decrease in the value of the respective confidence parameter 

30 on its absolute quality, the "absolute quality" of the 

confidence parameter being understood as its level . With a 
high confidence parameter, for example, a value increase can 
be correspondingly small so as not to create too great a 
difference between the confidence parameters, since too great 

35 a difference would otherwise interfere with an inherently 
correct correction of a misclassif ication . 
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This problem is also solved in another refinement of the 
method in which the permissible value range for the confidence 
parameters is limited. 

5 

An additional improvement to the example method according to 
the present invention is represented by weighted inclusion of 
the absolute quality of the respective individual 
classification results, with reference to the individual 
10 object classes, into the increase and/or decrease of the 

corresponding confidence parameter. The reliability of the 
individual results for each object class is thus also included 
in the classification result. A higher classification quality 
is the consequence. 

15 

A further advantageous method step is that of evaluative 
analysis of the calculated confidence parameters to determine 
a final, detailed classification result. It is thereby 
possible to arrive at more-differentiated statements regarding 
20 classification, for example the statement that the object fits 
badly into class A and fits very badly into all other classes. 
This is an improvement over the statement that "class A is the 
best fit." 

2 5 Also advantageous are on the one hand an evaluation 

constituting a classification into a higher-order class in the 
event the classification results alternate between similar 
object classes (example: if the result alternates between 
"small car" and "large car," the final result might be the 

30 higher-order class "car"), and on the other hand an evaluation 
constituting a refusal to classify if the classification 
results jump back and forth over time between dissimilar 
object classes (example: if the result alternates between 
"truck" and "pedestrian, 11 the final result might be "object 

35 cannot be reliably classified"). 
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The method according to the present invention can be used, in 
particular, to evaluate the classification results of a 
classification method for objects in the surroundings of a 
vehicle. This can be accomplished using a computer-assisted 
5 vehicle information system that contains the following 
devices : 

- effective connections to relevant interfaces to vehicle 
sensor devices for sensing objects in the vehicle 1 s 

10 surroundings ; 

- a control circuit for analysis and classification of the 
sensed objects, suitable for carrying out the method according 
to the present invention; and 

15 

- effective connections to relevant interfaces to actuator 
devices on the vehicle. 

BRIEF DESCRIPTION OF THE DRAWINGS 
20 Example embodiments of the present invention are explained 
below with reference to the figures. 

Figure 1 is a block depiction of a computer-assisted vehicle 
information system set up to carry out an example embodiment 
25 of the method according to the present invention. 

Figure 2 is a flow chart for an example embodiment of the 
method according to the present invention. 

3 0 Figure 3 is a sketch of a scene in the surroundings of a 

vehicle, with results from an example embodiment of the method 
according to the present invention. 

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS 
35 In the Figures, identical reference characters denote 
identical or identically functioning components. 
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Referring to Figure 1, sensors 5 for sensing objects (e.g. 
trees, pedestrians, passenger cars, or trucks) in the 
surroundings of vehicle 1 are mounted in and/or on vehicle 1, 
5 for example a passenger car. The nature of sensors 5 can be, 
for example, radar, ultrasound, video in the visible and 
infrared region, lidar, laser, and range imager camera. 

The measured data acquired by the sensors regarding a sensed 
10 object are forwarded via effective connections and interfaces 
(e.g. cable, radio, infrared transmission) to vehicle 
information system 3 integrated into vehicle 1. Vehicle 
information system 3 includes a computer unit 7 associated 
with which is a database 9 that contains data about features 
15 regarding object classes such as pedestrians, transport 
vehicles, and passenger cars. 

Also part of computer unit 7 is a control circuit 11 for 
classifying the sensed objects and evaluating the 

20 classification result. Both the measured data from sensors 5 

and the data about the features of the object classes required 
from database 9 are fed into this control circuit 11, and 
processed therein for classification. According to an example 
embodiment of the present invention, the confidence 

25 parameters, which indicate how well the sensed object matches 
the particular object class, are correspondingly increased or 
decreased in value depending on whether the previous 
classification is or is not confirmed, e.g., raised by three 
points per iteration, or lowered by three points for a 

3 0 decrease in value. 

According to an example embodiment of the present invention, a 
programmed evaluative analysis of the classification result 
also occurs in computer unit 7. This results in a 
35 differentiated classification, for example the statement that 
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the object fits very well into class A and fits moderately 
well into all other classes. 

When the classification result has achieved sufficient 
5 stability over time according to an example embodiment of the 
present invention, specific actuator devices 13 of vehicle 1 
can be triggered by control unit 7, if desired, via effective 
connections (e.g. cable, radio, infrared transmission); for 
instance, the brake can be actuated if the classification 
10 result is used, for example, in conjunction with a collision 
prediction and a safety system. 

Figure 2 depicts an example embodiment of the method according 
to the present invention as a flow chart. 

15 

A preclassif ication 100, which is based on a single 
measurement using the aforementioned sensors and a single 
interpretation of the measurement results obtained therefrom, 
and does not yet contain any stabilization over time and does 

20 not yet represent a confidence parameter described herein, 
supplies, for each object to be investigated, parameter 
numbers constituting a "quality parameter" for matching to all 
the classes that are to be distinguished. Motivated by 
information theory, what is used in this example as the 

25 parameter is a so-called word length, which should be short in 
the event of a good agreement between object and class. This 
"classification word length" is referred to hereinafter as 
"w_i," where i is the index for a class. 

30 As a possible enhancement, a "rejection word length" w_rej_i 
can be calculated in addition to the classification word 
lengths w_i . This parameter refers to the question of whether 
the object does NOT fit into the particular object class and 
is to be rejected. The calculation of w_rej_i can proceed, in 

35 principle, differently from that of w_i . Both 

preclassif ication results (w__i and w_rej_i) , or conversely 
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only one of the two, are used in this example as input 
variables of the example method according to the present 
invention; i.e., the aforementioned "quality parameter" serves 
for calculation of the actual confidence parameter, as 
5 described in more detail below. 

For the object to be classified, a so-called hypothesis exists 
for each of the object classes to be distinguished. Hypothesis 
i is "the object belongs to class i." 

10 

The probability that a hypothesis is true is evaluated using 
the confidence parameter "conf_i." The method according to the 
present invention uses these confidence parameters to evaluate 
all the classification word lengths. 

15 

In addition to the input data mentioned above, according to 
the present invention the evaluation system uses the 
confidence parameters from previous time steps (k-1) , i.e. 
"conf_i_k-l," for each of the individual hypotheses. 

20 

In processing step 110 (adaptation of the confidence 
parameters), a distinction 120 as to specific cases is made. 
These cases can be handled differently depending on the 
application, i.e., the system can in principle also be used 
25 without a distinction of cases, or with fewer or more 

distinctions. In this example the following three cases are 
distinguished : 

a) The object was classified in the current time step, and 
30 not rejected. 

b) The object was categorized by the classifier in the 
current time step as belonging to none of the object 
classes, and therefore rejected. This can be done, for 

35 example, by comparing the rejection word lengths w_rej_i 

for all the classes with a defined rejection threshold. 
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c) No classification word lengths (w_i and/or w_rej_i) exist 
for the object in the current time step. This can be 
attributable to insufficient classifier input data, but 
5 can also result from deliberate lack of processing of the 

object. It may be useful, for example, in the interest of 
high processing speed, not to reclassify in the current 
time step an object that already has a high confidence 
parameter, but instead to perform the classification only 
10 on the basis of the old confidence parameters. 

Case a) - The object was classified in the current time step : 

According to the classification word lengths w_i , an increase 
15 in the value of the corresponding confidence parameter is 

performed. One possibility is to increase the value according 
to a quality sort of the individual hypotheses: all the word 
lengths w__i are sorted in ascending order, and the smallest 
w_i indicates class that "best matches" the object. The 

2 0 confidence parameters for each hypothesis are increased in 

value in accordance with this ranked list. The formula might 
be as follows: 

confidence parameter for the class in first place on the 
25 list is increased in value by 5 points; 

confidence parameter for the class in second place on the 
list is increased in value by 3 points; 

3 0 - confidence parameter for the class in third place on the 

list is increased in value by 1 point; 

all other confidence parameters remain unchanged. 
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The number of classes to be taken into account, and the 
relationship among the value increases, can be selected 
without restriction based on the particular application. 

5 The increase in value is additionally performed as a function 
of the absolute quality of the individual result. Here an 
additional weighting factor g__i is calculated for each class 
(see step 130) . This factor takes into account the quality of 
the result for each class (i.e. the level of w_i in relation 
10 to a specific threshold) . The distance from w_rej_i to the 
rejection threshold may be used here as an example: this 
distance is an indication of the evaluation "the object does 
not belong to any of the classes." 

15 g_i = max (0; 1 - (w_rej_i / rejection threshold)) 

For objects that were classified but almost rejected, the 
weighting factor assumes a value close to 0. This means that 
the corresponding hypothesis was in fact barely confirmed, and 
20 an increase in the value of the confidence parameter is 
therefore not plausible. For a higher-quality result 
(corresponding to a greater distance from the rejection 
threshold), g_i approaches 1. This applies to the assumption 
of a positive w_rej_i. 

25 

The result for the increase in value of the confidence 
parameters is then: 5 * g_i for the class in first place, and 
3 * g_i and 1 * g_i correspondingly for second and third 
place , respectively . 

30 

Case b) - The object was "rejected" by the classifier in the 
current time step : 

In the event of rejection, the confidence parameters of all 
35 the hypotheses are decreased in value: a rejection of the 

object means that the object cannot be allocated to any of the 
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defined classes, and therefore none of the defined hypotheses 
can be supported. 



Correlated with the quantitative example of a 5, 3, 1 value 
5 increase cited in case a) , a value decrease of, for example, 4 
for all confidence parameters can be applied in case b) . 

As in case a) , a quality-dependent additional weighting factor 
g2_i is used for the value decrease in the confidence 
10 parameters for each class i (130) . This factor can once again, 
for example, be based on the distance between the rejection 
word length w_rej_i and a defined rejection threshold: 

g2_i = min (1; [1 - (w_rej_i / rejection threshold)]) 

15 

In this case g2_i assumes large values (a maximum of 1) for a 
large distance from the rejection threshold: the more 
unequivocally the object was rejected, the more the value of 
the confidence parameter must be decreased. For a w__rej_i 
20 close to the rejection threshold, weighting factor g2_i 
approaches 0 (no value decrease) . 

Case c) - No classification result exists for the object in 
the current time step : 

25 

If the classification word lengths (w_i and/or w_rej_i) are 
absent, all the confidence parameters are decreased in value 
by a specific identical amount. This represents a reduction in 
the certainty that the hypotheses are correct . For the 

30 quantitative example being used, a decrease in the value of 
all confidence parameters by 1 is selected. It is obvious in 
this context that the absolute numbers should be coordinated 
with the classification iteration rate, the value range 
limits, and the application of the method. The iteration rate 

35 here is the frequency with which individual quality parameters 
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are calculated, per predefined unit of time, in the aforesaid 
time steps k. 

Here as well, it is advantageous to apply a more 
5 differentiated value decrease as a function of various cases 
(e.g., by asking "why was the object not classified," etc.). 

Adapting the value range (see step 140) : 

10 An adaptation 140 of the value range of the confidence 

parameters is effected by limiting them. Here as well, the 
absolute numbers should be coordinated with the classification 
iteration rate and the application of the method. 

15 The value range does not necessarily have to be symmetrical 

about zero; a smaller range for negative confidence parameters 
and a larger range for positive ones may prove useful 
depending on the application. In this example a range from -2 0 
to 200 is selected. 

20 

If the confidence parameters depart from this value range as a 
consequence of an increase or decrease in value, an adaptation 
is necessary; otherwise a classification would no longer be 
possible, since all (or at least several) of the hypotheses 

2 5 would be supported to the same extent. This can be done 

differently for a value range overflow and underflow. One 
possible adaptation is: 

A value range underflow causes the confidence values to 
30 be truncated. If a confidence parameter is on the verge 

of dropping below this level, it is set to the minimum 
possible value. 

A value range overflow causes the confidence parameters 

3 5 not to be simply truncated, but to be adapted in 

accordance with their relationship to one another. When a 
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confidence parameter reaches the maximum value, for 
example, a value decrease can be performed for all the 
other confidence parameters by the same amount by which 
the high confidence parameter would normally be increased 
5 in value. The distance between the high confidence 

parameters and the other confidence parameters is thereby 
maintained . 

The aforesaid adaptation is only an example of the plurality 
10 of possible procedures. 

Determining the final classification result (step 150) : 

According to the present invention, a determination 150 of the 
15 classification result for the object in question is performed 
as the last processing step of the method according to the 
example embodiment. This result is ascertained on the basis of 
the calculated confidence parameters. 

20 A very large number of possibilities exist here for detailed 
evaluation, for example: 

Assign the object to the class for which the highest 
confidence parameter exists. 

25 

As an additional alternative: the classification result 
is valid only if this highest confidence parameter 
exceeds a given minimum threshold. 

30 - As an additional alternative: the classification result 
is valid only if the distance between this highest 
confidence parameter and the second-highest confidence 
parameter exceeds a defined threshold (otherwise the 
result is characterized as "not unequivocal"). 

35 
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For stabilization over time according to the present 
invention, the confidence parameters are temporarily stored 
and are used as input data for the next processing step of the 
method. 

5 

Figure 3 depicts a scene in the surroundings of a vehicle, 
showing the results of an example embodiment of the method 
according to the present invention. For each of the objects 
present in the scene (vehicles only, in this case) , one of 
10 which is labeled with reference character 206, icons 200 show 
the class of the object by way of a symbol 2 02, and indicate 
the level of the confidence parameters using a bar 204. This 
depiction is not intended for the driver, but serves only for 
further explanation of the present invention. 

15 

Although the present invention has been described above on the 
basis of a preferred example embodiment, it is not limited 
thereto but rather is modifiable in numerous ways, e.g., the 
absolute magnitude of the value increase/decrease delta 
20 numbers, the iteration duration, or the limits. 

The method according to the present invention can be used to 
improve any type of classification. The method encompasses a 
plurality of steps and variants that for the most part are 
25 mutually independent, and it can be flexibly configured. 

Lastly, the features of the present invention can be combined 
with one another substantially without restriction. 
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